Designing Interfaces for Human-Computer Communication: An On-Going Collection of Considerations

While we do not always use words, communicating what we want to an AI is a conversation -- with ourselves as well as with it, a recurring loop with optional steps depending on the complexity of the situation and our request. Any given conversation of this type may include: (a) the human forming an intent, (b) the human expressing that intent as a command or utterance, (c) the AI performing one or more rounds of inference on that command to resolve ambiguities and/or requesting clarifications from the human, (d) the AI showing the inferred meaning of the command and/or its execution on current and future situations or data, (e) the human hopefully correctly recognizing whether the AI's interpretation actually aligns with their intent. In the process, they may (f) update their model of the AI's capabilities and characteristics, (g) update their model of the situations in which the AI is executing its interpretation of their intent, (h) confirm or refine their intent, and (i) revise their expression of their intent to the AI, where the loop repeats until the human is satisfied. With these critical cognitive and computational steps within this back-and-forth laid out as a framework, it is easier to anticipate where communication can fail, and design algorithms and interfaces that ameliorate those failure points

Leveraging Learners for Teaching Programming and Hardware Design at Scale

In a massive open online course (MOOC), a single pro-gramming or digital hardware design exercise may yield thousands of student solutions that vary in many ways, some superï¬ cial and some fundamental. Understanding large-scale variation in student solutions is a hard but important problem. For teachers, this variation can be a source of pedagogically valuable examples and expose corner cases not yet covered by autograding. For students, the variation in a large class means that other students may have struggled along a similar solution path, hit the same bugs, and can offer hints based on that earned expertise. We developed three systems to take advantage of the solu-tion variation in large classes, using program analysis and learnersourcing. All three systems have been evaluated using data or live deployments in on-campus or edX courses with thousands of students

The Usability of Pragmatic Communication in Regular Expression Synthesis

Programming-by-example (PBE) systems aim to alleviate the burden of programming. However, user-specified examples are often ambiguous, leaving multiple programs to satisfy the specification. Consequently, in most prior work, users have had to provide additional examples, particularly negative ones, to further constrain the search over compatible programs. Recent work resolves additional ambiguity by modeling program synthesis tasks as pragmatic communication, showing promising results on a graphics domain using a rudimentary user-study. We adapt pragmatic reasoning to a sub-domain of regular expressions and rigorously study its usability as a means of communication both with and without the ability to provide negative examples. Our user study (N=30) demonstrates that, with a pragmatic synthesizer, end-users can more successfully communicate a target regex using positive examples alone (95%) compared to using a non-pragmatic synthesizer (51%). Further, users can communicate more efficiently (57% fewer examples) with a pragmatic synthesizer compared to a non-pragmatic one

Feature engineering for clustering student solutions

Open-ended homework problems such as coding assignments give students a broad range of freedom for the design of solutions. We aim to use the diversity in correct solutions to enhance student learning by automatically suggesting alternate solutions. Our approach is to perform a two-level hierarchical clustering of student solutions to first partition them based on the choice of algorithm and then partition solutions implementing the same algorithm based on low-level implementation details. Our initial investigations in domains of introductory programming and computer architecture demonstrate that we need two different classes of features to perform effective clustering at the two levels, namely abstract features and concrete features

Foobaz: Variable Name Feedback for Student Code at Scale

Current traditional feedback methods, such as hand-grading student code for substance and style, are labor intensive and do not scale. We created a user interface that addresses feedback at scale for a particular and important aspect of code quality: variable names. We built this user interface on top of an existing back-end that distinguishes variables by their behavior in the program. Therefore our interface not only allows teachers to comment on poor variable names, they can comment on names that mislead the reader about the variable's role in the program. We ran two user studies in which 10 teachers and 6 students created and received feedback, respectively. The interface helped teachers give personalized variable name feedback on thousands of student solutions from an edX introductory programming MOOC. In the second study, students composed solutions to the same programming assignments and immediately received personalized quizzes composed by teachers in the previous user study.Massachusetts Institute of Technology (Amar Bose Teaching Fellowship)Quanta Computer (Firm) (Qmulus Project

OverCode: Visualizing Variation in Student Solutions to Programming Problems at Scale 7:35

In MOOCs, a single programming exercise may produce thousands of solutions from learners. Understanding solution variation is important for providing appropriate feedback to students at scale. The wide variation among these solutions can be a source of pedagogically valuable examples and can be used to refine the autograder for the exercise by exposing corner cases. We present OverCode, a system for visualizing and exploring thousands of programming solutions. OverCode uses both static and dynamic analysis to cluster similar solutions, and lets teachers further filter and cluster solutions based on different criteria. We evaluated OverCode against a nonclustering baseline in a within-subjects study with 24 teaching assistants and found that the OverCode interface allows teachers to more quickly develop a high-level view of students' understanding and misconceptions, and to provide feedback that is relevant to more students' solutions

A Quadratic Regulator-Based Heuristic for Rapidly Exploring State Space

Kinodynamic planning algorithms like Rapidly-Exploring Randomized Trees (RRTs) hold the promise of finding feasible trajectories for rich dynamical systems with complex, nonconvex constraints. In practice, these algorithms perform very well on configuration space planning, but struggle to grow efficiently in systems with dynamics or differential constraints. This is due in part to the fact that the conventional distance metric, Euclidean distance, does not take into account system dynamics and constraints when identifying which node in the existing tree is capable of producing children closest to a given point in state space. We show that an affine quadratic regulator (AQR) design can be used to approximate the exact minimum-time distance pseudometric at a reasonable computational cost. We demonstrate improved exploration of the state spaces of the double integrator and simple pendulum when using this pseudometric within the RRT framework, but this improvement drops off as systems' nonlinearity and complexity increase. Future work includes exploring methods for approximating the exact minimum-time distance pseudometric that can reason about dynamics with higher-order terms